Apparatus and method for using a remote control system in surgical procedures

ABSTRACT

A system and method for using a remote control to control an electrosurgical instrument, where the remote control includes at least one momentum sensor. As the surgeon rotates their hand mimicking movements of a handheld electrosurgical instrument, the movements are translated and sent to the remote controlled (RC) electrosurgical instrument. The surgeon uses an augmented reality (AR) vision system to assist the surgeon in viewing the surgical site. Additionally, the surgeon can teach other doctors how to perform the surgery by sending haptic feedback to slave controllers. Also, the surgeon can transfer control back and forth between the master and slave controller to allow a learning surgeon to perform the surgery, but still allow the surgeon to gain control of the surgery whenever needed. Also, the surgeon could be located at a remote location and perform the surgery with the assistance of the AR vision system.m.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation and claims priority to U.S. patent applicationSer. No. 13/205,889, filed Aug. 9, 2011, the entire contents of whichare hereby incorporated by reference.

BACKGROUND

Technical Field

The present disclosure relates to a system and method for remotelycontrolling an electrosurgical instrument and, more particularly, to aremote control that uses momentum sensors to allow the surgeon to rotateand/or move the remote in a similar manner to handheld electrosurgicalinstrument.

Background of Related Art

Minimally invasive surgical procedures typically employ small incisionsin body cavities for access of various surgical instruments, includingforceps, laparoscopes, scalpels, scissors, and the like. It is often thecase that several surgical hands, such as several laparoscopicinstrument and camera holders, are necessary to hold these instrumentsfor the operating surgeon during the particular surgical procedure. Withthe introduction of robotic-assisted minimally invasive surgery (MIS) inrecent years, hospitals worldwide have made significant inves Intents inacquiring this latest technology for their respective facilities.

Thus, it is known to use robotic-assisted MIS when carrying out surgicaloperations. When surgery of this kind is performed, access to asubcutaneous surgical site is provided via a number (typically 3 to 5)of small (typically 5-12 mm) incisions, through which a surgical arm ismanually passed. The surgical arms are then coupled to the surgicalrobotic instrument, which is capable of manipulating the surgical armsfor performing the surgical operations, such as suturing or thermallycutting through tissue and cauterizing blood vessels that have beensevered. The surgical arms thus extend through the incisions during thesurgery, one of which incisions is used for supplying a gas, inparticular carbon dioxide, for inflating the subcutaneous area and thuscreate free space at that location for manipulating the surgicalinstruments.

Therefore, open surgeries often require a surgeon to make sizableincisions to a patient's body in order to have adequate visual andphysical access to the site requiring treatment. The application oflaparoscopy for performing procedures is commonplace. Laparoscopicsurgeries are performed using small incisions in the abdominal wall andinserting a small endoscope into the abdominal cavity and transmittingthe images captured by the endoscope onto a visual display. The surgeonmay thus see the abdominal cavity without making a sizable incision inthe patient's body, reducing invasiveness and providing patients withthe benefits of reduced trauma, shortened recovery times, and improvedcosmetic results. In addition to the endoscope, laparoscopic surgeriesare performed using long, rigid tools inserted through incisions in theabdominal wall.

However, conventional techniques and tools for performing laparoscopicprocedures may limit the dexterity and vision of the surgeon. Given thesize of the incisions, the maneuverability of the tools is limited andadditional incisions may be required if an auxiliary view of thesurgical site is needed. Thus, robotic instruments may be used toperform laparoscopic procedures.

One example of a robotic assisted MIS system is the da Vinci® Systemthat includes an ergonomically designed surgeon's console, a patientcart with four interactive robotic arms, a high performance visionsystem, and instruments. The da Vinci it console allows the surgeon tosit while viewing a highly magnified 3D image of the patient's interiorsent from the high performance vision system. The surgeon uses mastercontrols on the console that work like forceps to perform the surgery.The da Vinci® system corresponds to the surgeon's hand, wrist, andfinger movements into precise movements of the instruments within thepatient's interior.

However, the da Vinci® system only allows a single user to use theconsole and controllers at one time. Additionally, the 3D image shown inthe da Vinci® system can only be viewed by the surgeon sitting at theconsole which prevents other surgeon's from assisting the surgeon indetermining the best procedure to perform the surgery or from showingstudents how to perform the surgery. Additionally, the da Vinci® systemis large and cumbersome and oversized relative to the electrosurgicalinstruments used in the procedure.

SUMMARY

In accordance with the present disclosure, a system and method for usinga remote control to control an electrosurgical instrument, where theremote control includes at least one momentum sensor. As the surgeonrotates their hand mimicking movements of a handheld electrosurgicalinstrument, the movements are translated and sent to the remotecontrolled (RC) electrosurgical instrument. The surgeon uses anaugmented reality (AR) vision system to assist the surgeon in viewingthe surgical site. Additionally, the surgeon can teach otherdoctors/students how to perform the surgery by sending haptic feedbackto slave controllers. Also, the surgeon can transfer control back andforth between the master and slave controllers to teach another surgeonhow to perform the surgery, but still allow the teaching surgeon to gaincontrol of the surgery whenever needed. Also, the teaching surgeon couldbe located at a remote location and perform the surgery with theassistance of the AR vision system.

According to an embodiment of the present disclosure, a method forperforming an electrosurgical procedure includes the steps of generatinga pre-operative image of an anatomical section of a patient andanalyzing the pre-operative image to generate data about the anatomicalsection of the patient to assist a user during surgery. The method alsoincludes the steps of receiving a real time video signal of a surgicalsite within the patient and displaying on a user interface the analyzeddata with the video signal. Further, the method includes the step ofinserting a remote controlled electrosurgical instrument within thepatient. The remote controlled electrosurgical instrument is configuredto communicate with a remote. Additionally, the method includes thesteps of moving, rotating, and/or selecting a button on the remote in asimilar manner to a hand-held electrosurgical instrument while the useruses the user interface to view the surgical site, and moving, rotating,and/or performing other actions by the remote controlled surgicalinstrument based on movements and or actions of the remote.

According to another embodiment of the present disclosure, a method forperforming an electrosurgical procedure includes the step of inserting aremote controlled electrosurgical instrument within a patient. Theremote controlled electrosurgical instrument with at least one sensorand a base. The method further includes the step of moving a remote in amanner substantially similar to movement of a handheld electrosurgicalinstrument. The remote is configured with at least one momentum sensor.Also, the method includes the step of sending information from themomentum sensor to the base to move the remote controlledelectrosurgical instrument within the patient based on movements of theremote.

According to another embodiment of the present disclosure, a system forperforming an electrosurgical procedure includes a remote controlledelectrosurgical instrument configured to be inserted within a patient.The remote controlled electrosurgical instrument includes a base and thebase connects to a remote. The remote is configured with at least onemomentum sensor and at least one switch. The remote controlledelectrosurgical instrument responds to twists, rotation, side to side,up and down, diagonal, and other motions of the remote as a user movesthe remote in a similar manner to a handheld electrosurgical instrument.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present disclosure are described herein withreference to the drawings wherein:

FIG. 1 is a schematic diagram of a remote controlled surgical system inaccordance with an embodiment of the present disclosure;

FIGS. 2A-C are perspective views of different remotes used in accordancewith an embodiment of the present disclosure;

FIG. 3 is a perspective view of a remote controlled electrosurgicalinstrument in accordance with an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of an electrosurgical instrument controlsystem in accordance with an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of an augmented controller system inaccordance with an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of an augmented controller system inaccordance with an embodiment of the present disclosure;

FIG. 7 is a flow diagram of a process for controlling an electrosurgicalinstrument with a remote in accordance with an embodiment of the presentdisclosure;

FIG. 8 is a flow diagram of a process for determining if the remotecontrolled electrosurgical instrument is within an augmented safety zonein accordance with an embodiment of the present disclosure;

FIG. 9 is a schematic diagram of a master/slave remote system inaccordance with an embodiment of the present disclosure; and

FIG. 10 is a flow diagram of a process for sharing control of anelectrosurgical instrument between master and slave remotes inaccordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

Particular embodiments of the present disclosure are describedhereinbelow with reference to the accompanying drawings. In thefollowing description, well-known functions or constructions are notdescribed in detail to avoid obscuring the present disclosure inunnecessary detail.

FIG. 1 is a schematic diagram of a remote controlled surgical system 100that allows a surgeon M to perform a surgical procedure on patient Pusing a remote 200. Access to a subcutaneous surgical site withinpatient P is provided via a number (typically 3 to 5) of small(typically 5-12 mm) incisions 15, through which at least one remotecontrolled (RC) electrosurgical instrument 10 is manually passed.Additionally, a camera 150 is inserted in at least one incision 15 togive the surgeon M a view of the surgical site. The video signal fromthe camera may be sent to an Augmented Reality (AR) controller 600 (SeeFIGS. 5 and 6) to add additional data. The video signal and additionaldata are then displayed on a user interface 140. The AR displayed image142 may include labels on instruments, labels and/or margins of organs,tumors, or other anatomical bodies, and/or boundary zones arounddelicate anatomical bodies. The AR displayed image 142 may be in 2D or3D. As the camera 150 is moved around the surgical site, the labels anddata overlaid onto the video image move to the appropriate location.

The surgeon M controls the RC electrosurgical instrument 10 by rotatingand/or moving the remote 200 up, down, left, right, diagonally, and/orrotating. The movement of the remote 200 may be configured to move in amanner similar to a hand-held electrosurgical instrument. Additionally,the surgeon M can press a button on the remote 200 to activate anelectrical signal to coagulate or cut tissue, staple tissue, or performanother function of the instrument. The surgeon M can be located in thesame room as a patient or in a remote location such as another state orcountry. The remote 200 may be configured to send data to a base 300attached to the RC electrosurgical instrument 10. The data may be sentto the base 300 through a direct electrical connection or by Bluetooth®,ANT3®, KNX®, ZWave®, X10® Wireless USB®, IrDA®, Nanonet, Tiny OS®,ZigBee®, 802.11 IEEE, and other radio, infrared, UHF, VHF communicationsand the like.

FIGS. 2A-C show three possible embodiments of remote 200, however, otherembodiments may be possible. FIG. 2A discloses a first embodiment of aremote 220 that is generally circular in shape with a triangular frontthat may interconnect with the base 300 of the RC electrosurgicalinstrument 10. The circular shape allows the remote 220 to fit into thepalm of the surgeon's M hand, where the surgeon M can rotate his/herwrist to move the tool in a corresponding manner by easily pushing oneor more buttons 225, 227, 229, 231. The remote 220 includes at least onemomentum sensor 224 and an infrared sensor 222. The remote may beconfigured with one or more buttons 225, 227, 229, 231 that may belocated on the top, side, and/or bottom of the remote. Button 225 may beused to activate an electrical signal to coagulate or cut tissue, stapletissue, or perform other surgical functions. For example, button 227 maybe used to move the end effector assembly 100 in very small increments.Additionally, the remote 220 includes a haptic feedback mechanism 232that provides feedback about position, force used, instruction, andother similar uses. In an alternative embodiment, visual communicationmay be used to identify which instrument the remote is operating,problems with where the RC instrument 10 is located, battery life ofremote, which remote in a master/slave relationship is controlling theinstrument, and other problems with the RC instrument 10 or system.Alternatively, the remote 220 can be configured with audio feedback (notshown) to inform the surgeon M of problems or pre-recorded specificinstrument functions. The remote 220 further includes data ports 226 aand 226 b for communicating with the instrument base 300. The data ports226 a and 226 b may be connected directly to the instrument base 300 orwirelessly connected.

FIG. 2B discloses a second embodiment of a remote 240 for use with theremote controlled surgical system 100. Similar to the remote 220 in FIG.2A, the remote 240 includes data ports 226 a and 226 b, momentum sensor224, infrared sensor 222, and/or haptic feedback mechanism 232. Remote240 is shaped with a handle 245 and a trigger 244. The trigger 244 issimilar to button 225 on remote 220, and may be used to activate anelectrical signal to coagulate or cut tissue, staple tissue, or performanother surgical function. Remote 240 further includes buttons 227, 229,and 231 used to perform other functions of the RC instrument 10. Thesize and shape of the handle 245 can be ergonomically shaped for aright-handed or left-handed surgeon and/or based on the size of thesurgeon's hand.

FIG. 2C discloses a third embodiment of a remote 260. Similar to theremote 240 in FIG. 2B, the third remote 260 may include a housing 265, amomentum sensor 224, haptic feedback mechanism 232, handle 245, and/ortrigger 244. Trigger 244 is similar to button 225 on remote 220, and maybe used to activate an electrical signal to coagulate or cut tissue,staple tissue, or other procedure. Rotating wheel 262 is similar tobutton 227 on the first remote, and may be used to move the end effectorassembly 100 in very small increments. Data port 230 wirelessly connectsremote control 260 with the base 300 (see FIG. 3) of the RCelectrosurgical instrument 10. Similar to the second remote 240, thesize and shape of the handle 245 can be ergonomically shaped for aright-handed or left-handed surgeon and/or based on the size of thesurgeon's hand. In alternative embodiments, remote 260 may also includeopening 270 defined therein, where a surgeon M can insert the same typeend effector assembly 100 and shaft 12 as used within the patient Pduring surgery. This would allow the surgeon or others the ability seehow the end effector is moving.

Referring to FIG. 3, a RC surgical instrument 10, such as forceps,includes a shaft 12 that has a distal end 14 configured to mechanicallyengage an end effector assembly 100 operably associated with the forceps10 and a proximal end 16 that mechanically engages the base 300. In thedrawings and in the descriptions that follow, the term “proximal,” as istraditional, will refer to the end of the forceps 10 which is closer toa base 300, while the term “distal” will refer to the end that isfarther from the base. Alternatively, the system may be used with aremote controlled pencil or other electrosurgical instrument.

Drive assembly 130 is in operative communication with the remote 200through data port 340 for imparting movement of one or both of a pair ofjaw members 110, 120 of end effector assembly 100. The drive assembly130 may include a compression spring (not shown) or a drive wire 133 tofacilitate closing the jaw members 110 and 120 around pivot pin 111.Drive wire 133 is configured such that proximal movement thereof causesone movable jaw member, e.g., jaw member 120, and operative componentsassociated therewith, e.g., a seal plate 128, to move toward the otherjaw member, e.g., jaw member 110. With this purpose in mind, drive rodor wire 133 may be made from any suitable material and is proportionedto translate within the shaft 12. In the illustrated embodiments, drivewire 133 extends through the shaft 12 past the distal end 14. Both jawmembers 110 and 120 may also be configured to move in a bilateralfashion.

Base 300 receives an electrical signal from a generator (not shown).Generator may be connected to base 300 by a cable (not shown). By notincluding the generator within base 300, the size of base 300 may besmaller. Additionally, base 300 may be used with an existing generatorsystem. Alternatively, generator may be part of base 300.

Remote control 200 (See FIG. 3A) may be in operative communication withan ultrasonic transducer (not shown) via data port 340 when the RCsurgical instrument 10 is an ultrasonic instrument (not shown).Alternatively, base 300 may be arranged with multiple RC surgicalinstruments 10 attached. Each RC surgical instrument 10 may be removableor permanently attached to base 300.

FIG. 4 illustrates a control system 305 for the RC surgical instrument10 including the microcontroller 350 which is coupled to the positionand speed calculators 310 and 360, the loading unit identificationsystem 370, the drive assembly 130, and a data storage module 340. Inaddition, the microcontroller 350 may be directly coupled to a sensor315, such as a motion sensor, torque meter, ohm meter, load cell,current sensor, etc. The microcontroller 350 includes internal memorywhich stores one or more software applications (e.g., firmware) forcontrolling the operation and functionality of the RC surgicalinstrument 10.

The loading unit identification system 370 identifies to themicrocontroller 350 which end effector assembly 100 is attached to thedistal end 14 of the RC instrument 10. In an embodiment, the controlsystem 300 is capable of storing information relating to the forceapplied by the end effector assembly 100, such that when a specific endeffector assembly 100 is identified the microcontroller 350automatically selects the operating parameters for the RC surgicalinstrument 10. For example, torque parameters could be stored in datastorage module 320 for a laparoscopic grasper.

The microcontroller 350 also analyzes the calculations from the positionand speed calculators 310 and 360 and other sensors 315 to determine theactual position, direction of motion, and/or operating status ofcomponents of the RC surgical instrument 10. The analysis may includeinterpretation of the sensed feedback signal from the calculators 310and 360 to control the movement of the drive assembly 130 and othercomponents of the RC surgical instrument 10 in response to the sensedsignal. Alternatively, the location of the RC surgical instrument 10 maybe calculated using the method disclosed in U.S. Ser. No. 12/720,881,entitled “System and Method for Determining Proximity Relative to aCritical Structure” filed on Mar. 10, 2010, which is hereby incorporatedby reference. The microcontroller 350 is configured to limit the travelof the end effector assembly 100 once the end effector assembly 100 hasmoved beyond a predetermined point as reported by the positioncalculator 310. Specifically, if the microcontroller determines that theposition of the end effector assembly 100 is within a safety zonedetermined by the AR controller 200, the microcontroller is configuredto stop the drive assembly 130.

In one embodiment, the RC surgical instrument 10 includes varioussensors 315 configured to measure current (e.g., an ampmeter),resistance (e.g., an ohm meter), and force (e.g., torque meters and loadcells) to determine loading conditions on the end effector assembly 100.During operation of the RC surgical instrument 10 it is desirable toknow the amount of force exerted on the tissue for a given end effectorassembly 100. Detection of abnormal loads (e.g., outside a predeterminedload range) indicates a problem with the RC surgical instrument 10and/or clamped tissue which is communicated to the user.

The data storage module 320 records the data from the sensors 315coupled to the microcontroller 350. In addition, the data storage module320 may record the identifying code of the end effector assembly 100,user of surgical tool, and other information relating to the status ofcomponents of the RC surgical instrument 10. The data storage module 320is also configured to connect to an external device such as a personalcomputer, a PDA, a smartphone, or a storage device (e.g., a SecureDigital™ card, a CompactFlash card, or a Memory Stick™) through awireless or wired data port 340. This allows the data storage module 320to transmit performance data to the external device for subsequentanalysis and/or storage. The data port 340 also allows for “in thefield” upgrades of the firmware of the microcontroller 350.

Embodiments of the present disclosure may include an augmented reality(AR) control system 610 as shown in FIGS. 5-6. The RC surgicalinstrument 10 is connected to an AR controller 600 via the data port 660which may be either wired (e.g., FireWire , USB, Serial RS232, SerialRS485, USART, Ethernet, etc.) or wireless (e.g., Bluetooth®, ANT3®,KNX®, Z-Wave X10®, Wireless USB®, Wi-Fi , IrDA®, nanoNET®, TinyOS®,ZigBee®, 802.11 IEEE, and other radio, infrared, UHF, VHF communicationsand the like). Additionally, remote 200 (220, 240, 260) is connected tothe AR controller 600 via data port 660 which may be either wired (e.g.,FireWire®, USB, Serial RS232, Serial RS485, USART, Ethernet, etc.) orwireless (e.g., Bluetooth®, ANT3®, KNX®, Z-Wave , X10®, Wireless USB®,Wi-Fi®, IrDA®, nanoNET®, TinyOS®, ZigBee®, 802.11 IEEE, and other radio,infrared, UHF, VHF communications and the like).

FIG. 5 illustrates a schematic diagram of an AR control system 610 inaccordance with an embodiment of the present disclosure. With referenceto FIG. 5, the augmented reality (AR) controller 600 is configured tostore data transmitted to the controller 600 by a RC surgical instrument10 and a remote 200 (220, 240, 260) as well as process and analyze thedata. The RC surgical instrument 10 is a robotic instrument. The ARcontroller 600 is also connected to other devices, such as a videodisplay 140, a video processor 120 and a computing device 180 (e.g., apersonal computer, a PDA, a smartphone, a storage device, etc.). Thevideo processor 120 may be used for processing output data generated bythe AR controller 600 for output on the video display 140. Additionally,the video processor 120 may receive a real time video signal from acamera 150 inserted into the patient during the surgical procedure. Thecomputing device 180 may be used for additional processing of thepre-operative imaged data. In one embodiment, the results ofpre-operative imaging such as an ultrasound, MM, x-ray, or otherdiagnosing image may be stored internally for later retrieval by thecomputing device 180.

The AR controller 600 includes a data port 660 (FIG. 6) coupled to themicrocontroller 650 which allows the AR controller 600 to be connectedto the computing device 180. The data port 660 may provide for wiredand/or wireless communication with the computing device 180 providingfor an interface between the computing device 180 and the AR controller600 for retrieval of stored pre-operative imaging data, configuration ofoperating parameters of the AR controller 600 and upgrade of firmwareand/or other software of the AR controller 600.

Components of the AR controller 600 are shown in FIG. 6. The ARcontroller 600 includes a microcontroller 650, a data storage module 655a user feedback module 665, an OSD module 640, a HUD module 630, and adata port 660.

The data storage module 655 may include one or more internal and/orexternal storage devices, such as magnetic hard drives, or flash memory(e.g., Secure Digital® card, Compact Flash® card, or MemoryStick®). Thedata storage module 655 is used by the AR controller 600 to store datafrom the RC surgical instrument 10 and remote 200 (220, 240, 260) forlater analysis of the data by the computing device 180. The data mayinclude information supplied by a sensor 315 (FIG. 4), such as a motionsensor, torque sensor, and other sensors disposed within the RC surgicalinstrument 10.

The microcontroller 650 may supplant, complement, or supplement thecontrol circuitry 305 of the RC surgical instrument 10 shown in FIG. 4.The microcontroller 650 includes internal memory which stores one ormore software applications (e.g., firmware) for controlling theoperation and functionality of the RC surgical instrument 10. Themicrocontroller 650 processes input data from the computing device 180and adjusts the operation of the RC surgical instrument 10 in responseto the inputs. The RC surgical instrument 10 is configured to connect tothe AR controller 600 wirelessly or through a wired connection via adata port 340. The microcontroller 650 is coupled to the user feedbackmodule 665 which is configured to inform the user of operationalparameters of the RC surgical instrument 10. The user feedback module665 may be connected to a user interface. The user feedback module 665may be coupled to the haptic mechanism 232 within the remote 200 (220,240, 260) to provide for haptic or vibratory feedback. The hapticfeedback may be used in conjunction with the auditory and visualfeedback or in lieu of the same to avoid confusion with the operatingroom equipment which relies on audio and visual feedback. The hapticmechanism 232 may be an asynchronous motor that vibrates in a pulsatingmanner. In one embodiment, the vibrations are at a frequency of about 30Hz or above. The haptic feedback can be increased or decreased inintensity. For example, the intensity of the feedback may be used toindicate that the forces on the instrument are becoming excessive. Inalternative embodiments, the user feedback module 265 may also includevisual and/or audible outputs.

The microcontroller 650 outputs data on video display 140 and/or theheads-up display (HUD) 635. The video display 140 may be any type ofdisplay such as an LCD screen, a plasma screen, electroluminescentscreen and the like. In one embodiment, the video display 140 mayinclude a touch screen and may incorporate resistive, surface wave,capacitive, infrared, strain gauge, optical, dispersive signal oracoustic pulse recognition touch screen technologies. The touch screenmay be used to allow the user to provide input data while viewing ARvideo. For example, a user may add a label identifying the surgeon foreach tool on the screen. The HUD display 635 may be projected onto anysurface visible to the user during surgical procedures, such as lensesof a pair of glasses and/or goggles, a face shield, and the like. Thisallows the user to visualize vital AR information from the AR controller600 without loosing focus on the procedure.

The AR controller 600 includes an on-screen display (OSD) module 640 anda HUD module 630. The modules 640, 630 process the output of themicrocontroller 650 for display on the respective displays 140 and 635.More specifically, the OSD module 640 overlays text and/or graphicalinformation from the AR controller 600 over video images received fromthe surgical site via camera 150 (FIG. 1) disposed therein.Specifically, the overlaid text and/or graphical information from the ARcontroller 600 includes computed data from pre-operative images, such asx-rays, ultrasounds, MRIs, and/or other diagnosing images. The computingdevices 180 stores the one or more pre-operative images. In analternative embodiment, the data storage module 655 can store thepre-operative image. The AR controller 600 processes the one or morepre-operative images to determine margins and location of an anatomicalbody in a patient, such as an organ or a tumor. Alternatively, thecomputing device 180 can process and analyze the pre-operative image.Additionally, the AR controller can create safety boundaries arounddelicate structures, such as an artery or organ. Further, the ARcontroller 600 can decipher the one or more pre-operative images todefine structures, organs, anatomical geometries, vessels, tissueplanes, orientation, and other similar information. The AR controller600 overlays the information processed from the one or morepre-operative images onto a real time video signal from the camera 150within the patient. The augmented video signal including the overlaidinformation is transmitted to the video display 140 allowing the user tovisualize more information about the surgical site including areaoutside the vision of the camera 150. Additionally, as the camera movesaround the surgical site, the labels and/or data overlaid is moved tothe appropriate location on the real time video signal.

FIG. 7 is a flow diagram of a process 700 for controlling anelectrosurgical instrument with a remote 200 (220, 240, 260) accordingto an embodiment of the invention. After the process 700 starts at step705, a pre-operative image is generated from a diagnosing imagingsource, such as from an MRI, ultrasound, x-ray, CAT scan, etc. at step710. The pre-operative image is taken of an anatomical section of thepatient, which may include organs, tissue, vessels, bones, tumors,muscles, etc. Multiple images can be generated from one or more sourcesbased on the information required by the surgeon M. Next, thepre-operative image is analyzed to generate data to assist the surgeon Mduring surgery at step 715. The analyzing may be done by the computingdevice 180 or the microprocessor 650. The data may include margins andlocation of the anatomical section. Prior to starting the surgery, acamera 150 is inserted within the patient. A real time video signal ofthe patient during the surgical procedure is received at AR controller600 during the surgical procedure at step 720. The analyzed data isdisplayed with the real time video signal at step 725. For example, ifthe anatomical section is a tumor then the location and margins of thetumor are calculated and then the name and margins are augmented ontothe video signal to assist the surgeon M in locating the tumor. A RCelectrosurgical instrument 10 is inserted into a body cavity or incisionat step 730. A user M moves, twists, and/or selects buttons on theremote control 200 at step 735. The surgeon M may move the remote 200 ina manner similar to actions done with a handheld electrosurgicalinstrument. Before the process 700 ends at step 745, the RC surgicalinstrument 10 moves, twist, and/or performs other action based on themovements performed by the remote 200 at step 740. The movements of theremote 200 are sent wirelessly or the remote is directly connected tothe RC surgical instrument 10.

FIG. 8 is a flow diagram of process 800 for determining if the remotecontrolled electrosurgical instrument is within an augmented safety zoneaccording to an embodiment of the invention. After the process 800starts at step 805, a pre-operative image of an anatomical section of apatient is generated at step 810. The pre-operative image can begenerated from any type of diagnosing image, such as an x-ray, MRI, CATscan, ultrasound, etc. The pre-operative image analyzed to determine asafety zone around organs, tissue, and/or other delicate anatomicalstructures at step 815. Prior to starting the surgical procedure, acamera 150 is inserted within the patient. During the surgicalprocedure, a real time video signal is received by the AR controller 600via video processor 120 at step 825. The AR controller 600 augments thesafety zone onto the video signal at step 830. For example, the safetyzone may be represented as a cross hatched area or in a different color,such as a yellow area around an organ. A RC electrosurgical instrument10 is inserted into a cavity or incision 15 within the patient P at step830. The location of the surgical instrument 10 within the patient ismeasured at step 835 using the position calculator 310, speed calculator360, and other sensors 315. Alternatively, the location of the RCinstrument 10 and/or end effector assembly 100 may be calculated usingthe method disclosed in U.S. Ser. No. 12/720,881, entitled “System andMethod for Determining Proximity Relative to a Critical Structure” filedon Mar. 10, 2010, which is hereby incorporated by reference. The ARcontroller 200 determines if the RC surgical instrument 10 is within thesafety zone at step 840. If the RC surgical instrument 10 is not withinthe safety zone, then allow the surgeon M to move, twist, and/or selectbuttons on remote 200 at step 845. The movement of the remote may besimilar to movement of a handheld electrosurgical instrument. The RCelectrosurgical instrument 10 then moves, twists, and/or performs actionbased on the movement or actions from the remote at step 850. Then, thesystem measures the new location of the RC electrosurgical instrument 10at step 835. If the RC electrosurgical instrument 10 is within thesafety zone, then the AR controller 600 notifies the surgeon at step855. This notification can be visual, audible, or haptic feedback.Additionally, before process 800 ends at step 865, the AR controller600, if necessary, can stop the drive assembly 130 at step 860.

FIG. 9 illustrates a schematic diagram of a master/slave remote controlsystem 900 according to an embodiment of the invention. Similar to theremote controlled surgical system 100 shown in FIG. 1, the master/slaveremote control system 900 includes a patient P with at least oneincision 15, a RC electrosurgical instrument 10, a base 300, a camera150, and a display 140 with an augmented displayed image 142.Additionally, the master/slave remote control system 900 includes afirst user that is the master M (surgeon) that uses a master remote 960and at least one slave user 950 a, 950 b that uses a slave remote 970 a,b. As the master M moves, tilts, selects buttons on the master remote960, the slave remote may receive haptic feedback to teach the slaveuser how to move the slave remote 970 a. Additionally, the master remote960 may allow the master M to transfer control to slave remote 970 a andthen to 970 b or back to the master remote 960. The master M can belocated in the same room with slave 950 a and/or 950 b or the master Mcan be located in a remote location, such as another state or country.

FIG. 10 is a flow diagram of a process 1000 for sharing control of anelectrosurgical instrument between a master and a slave according to anembodiment of the invention. The process 1000 starts at step 1010,displaying a real time video signal with data from a pre-operative imageat step 1020. The master remote 960 selects a slave remote 970 a tocontrol the RC electrosurgical instrument at step 1030. The slave user950 a moves, twists, or selects buttons on the slave remote 970 a tocontrol the RC electrosurgical instrument 10. The master M may overridethe slave remote 970 a to regain control of the RC electrosurgicalinstrument at step 1040. Before the process 1000 ends at step 1060, themaster M moves, tilts, and/or selects buttons on the master remote 960,haptic feedback is sent to the slave remote 970 a, b at step 1050 totrain the slave user 950 a, b how to use the remote.

While several embodiments of the disclosure have been shown in thedrawings and/or discussed herein, it is not intended that the disclosurebe limited thereto, as it is intended that the disclosure be as broad inscope as the art will allow and that the specification be read likewise.Therefore, the above description should not be construed as limiting,but merely as exemplifications of particular embodiments. Those skilledin the art will envision other modifications within the scope and spiritof the claims appended hereto.

1. (canceled)
 2. A method for performing a surgical procedure, themethod comprising: mechanically connecting a wireless remote with aremote controlled surgical instrument such that the wireless remotedirectly contacts the remote controlled surgical instrument; and usingthe remote controlled surgical instrument as a handheld instrument toperform a surgical task while the remote controlled surgical instrumentis mechanically connected with the wireless remote.
 3. The methodaccording to claim 2, further comprising moving at least a portion ofthe remote controlled surgical instrument with respect to a patient. 4.The method according to claim 2, further comprising selecting a secondremote controlled surgical instrument using a user interface on thewireless remote.
 5. The method according to claim 2, further comprisingtransferring control of the remote controlled surgical instrument fromthe wireless remote to a slave remote.
 6. The method according to claim5, further comprising utilizing the wireless remote to transfer controlof the remote controlled surgical instrument from the slave remote tothe wireless remote.
 7. The method according to claim 2, furthercomprising training a user of a slave remote by sending feedback to theslave remote based on at least one of movements or actions of thewireless remote.
 8. The method according to claim 2, further comprisingtracking movement of the wireless remote with at least one momentumsensor of the wireless remote.
 9. The method according to claim 2,further comprising using at least one infrared sensor on the wirelessremote to facilitate mechanically connecting the wireless remote withthe remote controlled surgical instrument.
 10. The method according toclaim 2, further comprising tracking movement of the wireless remotealong x-, y- and z-axes using at least three momentum sensors of thewireless remote.
 11. The method according to claim 2, further comprisingselecting a feature on the wireless remote to activate a clampingfunction, to activate radio frequency energy for coagulation or cutting,or to activate a stapling function of the remote controlled surgicalinstrument.
 12. The method according to claim 2, further comprisingusing the wireless remote to perform a surgical task when the wirelessremote is free from mechanical contact with the remote controlledsurgical instrument.
 13. The method according to claim 2, furthercomprising mechanically engaging the remote controlled surgicalinstrument with at least one data port disposed at a distal position ofthe wireless remote when the wireless remote is mechanically connectedwith the remote controlled surgical instrument.
 14. The method accordingto claim 2, further comprising communicating data from the wirelessremote to the remote controlled surgical instrument.
 15. A method forperforming a surgical procedure, the method comprising: mechanicallyconnecting a master remote with a remote controlled surgical instrument;and using the remote controlled surgical instrument as a handheldinstrument to perform a surgical task while the remote controlledsurgical instrument is mechanically connected with the master remote.16. The method according to claim 15, further comprising moving at leasta portion of the remote controlled surgical instrument with respect to apatient, wherein the remote controlled surgical instrument includes atleast one of a sensor or a base.
 17. The method according to claim 16,further comprising moving the master remote in a manner substantiallysimilar to movement of the handheld surgical instrument, wherein themaster remote is wireless and includes at least one of a momentum sensoror a haptic feedback mechanism.